The various shapes of ferrite magnetic tiles and the wide varieties of their surface defects are great challenges for computer vision based surface defect quality recognition. To address this problem, the deep learning technique was introduced to the magnetic tile surface quality recognition, and a surface defect detection system for magnetic tiles was proposed based on convolution neural networks. Firstly, the tile target was segmented from the collected image and was rotated in order to obtain the standard image. After that, the improved multiscale ResNet18 was used as the backbone network to design the recognition system. During the training process, a novel within-class mixup operation was designed to improve the generalization ability of the system on the samples. To close to the practical application scenes, a surface defect dataset was built with the consideration of illumination changes and posture differences. Experimental results on the self-built dataset indicate that the proposed system achieves recognition accuracy of 97.9%, and provides a feasible idea for the automatic recognition of magnetic tile surface defects.